ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering

Yakun Song1, Zhuo Chen2, Xiaofei Wang2, Ziyang Ma1, Xie Chen1
1 Shanghai Jiao Tong University, Shanghai, China
2 Microsoft, One Microsoft Way, Redmond, USA

[Paper] [Bibtex] [code]

Abstract

The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation. However, existing methods still have some limitations: 1) repetitions, transpositions, and omissions in the output synthesized speech due to limited alignment constraints between audio and phoneme tokens; 2) challenges of fine-grained control over the synthesized speech with autoregressive (AR) language model; 3) infinite silence generation due to the nature of AR-based decoding, especially under the greedy strategy. To alleviate these issues, we propose ELLA-V, a simple but efficient LM-based zero-shot text-to-speech (TTS) framework, which enables fine-grained control over synthesized audio at the phoneme level. The key to ELLA-V is interleaving sequences of acoustic and phoneme tokens, where phoneme tokens appear ahead of the corresponding acoustic tokens. The experimental findings reveal that our model outperforms VALL-E in terms of accuracy and delivers more stable results using both greedy and sampling-based decoding strategies. The implementation of the algorithm is available at [code].

[*] This page is for research demonstration purposes only.

Zero-Shot TTS Continutaion Samples.

We perform the zero-shot continuation task on the LibriSpeech test-clean set. For each sample, we take the first 3 seconds as the audio prompt and make it generate the original text of the sentence (as the text prompt). 8 samples are randomly selected and shown below.

Text Prompt
Acoustic Prompt
Ground Truth
Ground Truth-Encodec
ELLA-V
VALL-E
Forthwith, all ran to the opening of the tent to see what might be amiss. But Master Will, who peeped out first, needed no more than one glance.
The undertaking, which seemed so attractive when viewed as a literary task, proved to be a most difficult one, and at times, it became oppressive.
Many laws exist among us which are the counterpart of yours as they were in the olden time.
I will briefly describe them to you, and you shall read the account of them at your leisure in the sacred registers.
Again, I thank you. This incident, I suppose, will be renewed no more. If I live to be an old woman, I shall remember it thirty years hence as a bright dream.
She saw that the bed was gilded and so rich that it seemed that of a prince rather than of a private gentleman.
Some points may be taken as fixed, and such as any theory of memory must arrive at.
Then she gave Rosalie back her magic ring, thanking the kind witch for all she had done for them.

Zero-Shot TTS on Hard Cases

The sentences of the proposed hard case set include challenging phonetic patterns, alliteration, and unusual (abnormal) combinations of words that might pose difficulties for a TTS system to generate natural-sounding speech.

We randomly selected the audio prompt from LibriSpeech test-clean set and randomly selected 8 generated samples. All samples are derived from our proposed Hard Case set. In the following, text prompt refers to the content that the model is asked to generate.

Text Prompt
Acoustic Prompt
Acoustic Prompt (Encodec)
ELLA-V
VALL-E
Playful pandas prattled playfully, playful playfully peaches.
Worried walruses worriedly waddled, worried worried waddles.
Silly siblings silently signaled silly signals, sparking silly scenarios.
In the timeless tapestry of the galaxy, the mesmerizing dance dance dance dance dance of celestial bodies unveiled the cosmic secrets of the universe.
Navigating through through through through through through the dense forest, the intrepid explorer discovered an ancient ancient ancient ancient ancient ancient relic buried beneath layers of thick underbrush.
Expressive eagles elegantly echoed expressive expressive echoes.
Happy hummingbirds happily hovered, happy happy hums.
The early bird gets the early bird gets the early bird gets the worm, but the second mouse gets the worm, but the second mouse gets the cheese.

100 Hard Sentences

  1. Active artists always appreciate artistic achievements and applaud awesome artworks.
  2. Brave bakers boldly baked big batches of brownies in beautiful bakeries.
  3. Clever cats carefully crafted colorful collages, creating cheerful compositions.
  4. Daring dancers dazzled during dynamic dance displays, drawing delighted crowds.
  5. Excited engineers eagerly enjoyed exploring enormous engineering exhibits.
  6. Friendly farmers faithfully fostered fields, favoring fruitful crops.
  7. Gallant gophers gracefully gambled golden gooseberries on grandiose glaciers.
  8. Happy hikers harmoniously hiked through hilly landscapes on heavenly holidays.
  9. Inquisitive individuals ingeniously invented innovative inventions.
  10. Jovial joggers joyfully joined jogging jaunts, justifying joyful jolliness.
  11. Keen kids keenly knitted knotted knots in kindergartens.
  12. Lively librarians lazily leafed through large literary collections.
  13. Merry musicians melodiously meandered through memorable melodies.
  14. Nifty nurses navigated nearby neighborhoods, noting necessary nuances.
  15. Optimistic office workers openly observed office occurrences, offering opinions.
  16. Playful puppies passionately played, producing playful puppy pandemonium.
  17. Quick quilters quietly quilted quality quilts, quickly quelling queries.
  18. Radiant readers radiantly recited rhythmic rhymes, receiving round applause.
  19. Silly siblings silently signaled silly signals, sparking silly scenarios.
  20. Tenacious teachers tirelessly taught tricky topics, taking time to tutor.
  21. Unwavering umpires upheld umpiring rules, understanding umpire responsibilities.
  22. Vivacious volunteers voraciously volunteered, validating valuable ventures.
  23. Witty writers wittily wrote whimsical words, weaving wonderful worlds.
  24. Xylophone xylophonists expertly executed exciting xylophone exhibitions.
  25. Young yogis yearned for yogic yearning, yielding youthful yawns.
  26. Zany zookeepers zealously zipped through zoos, zealously zeroing in on zebras.
  27. Adventurous ants anxiously ate apples, adventurous adventurous apples.
  28. Playful pandas prattled playfully, playful playfully peaches.
  29. Colorful cats cheerfully chased colorful colorful critters.
  30. Confused canines clumsily crunched confused confused cookies.
  31. Expressive eagles elegantly echoed expressive expressive echoes.
  32. Silly squirrels stumbled stumblingly, silly silly feathers.
  33. Happy horses happily hopped, happy happy hops.
  34. Clumsy cats carelessly caught clumsy clumsy butterflies.
  35. Mixed-up mice munched mixed-up mixed-up muffins.
  36. Curious koalas curiously climbed curious curious climbers.
  37. Chatty chipmunks chatted chattily, chatty chatty chats.
  38. Sad snakes sadly sighed sad sad sighs.
  39. Funny ferrets fumbled fumblingly, funny funny feathers.
  40. Excited elephants excitedly explored excited excited explorations.
  41. Confused crabs confusedly cracked confused confused crab cakes.
  42. Joyful jaguars joyfully jumped joyful joyful jumps.
  43. Lost lions lazily lounged lost lost landscapes.
  44. Noisy newts nonsensically nibbled noisy noisy nibbles.
  45. Eager otters eagerly overcame eager eager obstacles.
  46. Puzzled penguins puzzledly pondered puzzled puzzled ponderings.
  47. Quaint quokkas quietly questioned quaint quaint queries.
  48. Restless rabbits rambled restlessly, restless restless radishes.
  49. Happy hummingbirds happily hovered, happy happy hums.
  50. Crazy cats crazily cavorted crazy crazy cavorts.
  51. Worried walruses worriedly waddled, worried worried waddles.
  52. Xenial xenophobes xenially xeroxed xenial xenial xylophones.
  53. Yearning yaks yawned yearningly, yearning yearning yawns.
  54. Zany zebras zanily zigzag, zany zany zigzag.
  55. l l l l l l b v p d s s s s s l
  56. forty five to five hundred and eleven Fail - one to zero Cancelled - zero - zero Total
  57. two thousand two hundred twenty two happily happy two hundred and twenty-two
  58. g 2 p 2 g 2 p 2 g 2 p 2 p 2 p
  59. F 1 F 2 F 4 F 8 H 16 H 32 H 64
  60. Amidst the towering skyscrapers, the bustling cityscape echoed with the incessant hum hum hum hum hum of commerce and ambition.
  61. As the ancient ancient ancient ancient ancient ancient manuscript was deciphered, the archaeologist unearthed a trove of forgotten forgotten forgotten forgotten forgotten forgotten wisdom embedded in in in in in in the crumbling parchment.
  62. As the cosmic cosmic cosmic cosmic cosmic cosmic dance of the stars unfolds in in in in in in silence, revealing the mystical mysteries of the celestial celestial celestial celestial celestial celestial realm.
  63. As the quantum physicist delved into the quantum realm, the enigmatic entanglement of particles perplexed even the most astute astute astute astute astute astute minds.
  64. Beneath the celestial canopy, the twinkling stars whispered tales of timeless timelessness and cosmic conspiracies.
  65. Beneath the moonlit night, the solitary wolf’s haunting howl howl howl howl howl echoed through the ancient forest, embodying the primal spirit of the wilderness.
  66. Beneath the shimmering sheen of the silken fabric fascinated the fashionistas, creating a cacophony of compliments.
  67. Coding in in in in in in solitude, the programmer uncovered the intricate intricacies of the algorithm, unveiling a hidden hidden hidden layer of complexity.
  68. Crafting a symphony of flavors, the skilled chef orchestrated a culinary masterpiece that left an indelible mark mark mark mark mark on the palates of the discerning diners.
  69. Delving into the dense foliage, the intrepid explorer uncovered the cryptic trail trail trail trail trail leading to an undiscovered waterfall deep in the heart of the jungle.
  70. In the bustling bustling bustling bustling bustling bustling metropolis, the rhythmic rhythm of life beats to the pulsating pulsating pulsating pulsating pulsating heartbeat of the city streets.
  71. In the enigmatic enigmatic enigmatic enigmatic enigmatic enigmatic maze, the elusive solution remained hidden behind layers of intricate intricacies.
  72. In the realm of robotics, replicating the nuanced nuances of human emotions remains a formidable formidable formidable formidable formidable challenge.
  73. In the timeless tapestry of the galaxy, the mesmerizing dance dance dance dance dance of celestial bodies unveiled the cosmic secrets of the universe.
  74. In the vast vast vast vast vast vast landscape of knowledge, the voracious voracious voracious voracious voracious voracious reader explored the depths of literature, extracting pearls of wisdom hidden within within within within within within the pages.
  75. Navigating through through through through through through the dense forest, the intrepid explorer discovered an ancient ancient ancient ancient ancient ancient relic buried beneath layers of thick underbrush.
  76. Navigating through the labyrinthine labyrinthine labyrinthine labyrinthine labyrinthine labyrinthine code, the programmer discovered an elusive bug.
  77. Precision in decision-making is paramount, as an error could cost countless countless countless countless countless countless countless lives.
  78. Strategically strategizing strategy in the fast-paced world of e-sports demands a dexterous dexterous dexterous dexterous dexterous dexterous mind.
  79. The clandestine clandestine clandestine clandestine clandestine clandestine mission required a meticulous analysis of encrypted encryptions and covert communications.
  80. The entrepreneur, driven by an insatiable insatiable insatiable insatiable insatiable desire for success, embarked on a journey filled with challenges, challenges, challenges, challenges, challenges, challenges, and triumphs.
  81. The hypnotic sway sway sway sway sway of the pendulum clock echoed through the dimly lit room, creating an atmosphere of temporal contemplation.
  82. The relentless relentless relentless relentless relentless relentless pursuit of perfection in in in in in in in in in craftsmanship led the artisan to create an exquisite masterpiece admired for its meticulous meticulous meticulous meticulous meticulous meticulous details.
  83. The resounding echoes of the ancient drums resonated through the cavern, revealing the sacred rhythm rhythm rhythm rhythm rhythm of the hidden rituals.
  84. The shimmering sheen of the silken fabric fascinated the fashionistas, creating a cacophony of compliments.
  85. Theoretical thespians theatrically theorized thematic thesauri theses.
  86. Through the misty morning, the distant call call call call call of migratory birds echoed, heralding the arrival of the changing seasons.
  87. Within the labyrinthine corridors, the footsteps footsteps footsteps footsteps footsteps of the inquisitive investigator echoed, unraveling the enigma of the haunted mansion.
  88. A journey of a thousand miles begins begins begins begins begins with a single step. Step by step, step by step, step by step, progress is made.
  89. The impact of technology on society is society is society is society is society is profound. Profound Profound Profound Profound Consideration of consequences is imperative in technological innovation.
  90. The future belongs to belongs to belongs to belongs to belongs to those who believe in the beauty of the beauty of the beauty of the beauty of the beauty of their dreams.
  91. Silence is a source of great strength. Strength lies in lies in lies in lies in inner resilience.
  92. In the realm of possibilities, possibilities, possibilities, possibilities, possibilities, the only limitation is imagination.
  93. Learning from mistakes is is is is is is a crucial part of the journey to success.
  94. Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop not to stop not to stop not to stop not to stop questioning.
  95. Creativity is intelligence is intelligence is having fun. The only limit to our realization of tomorrow is our doubts of today.
  96. In the world of technology, the only constant is the only constant is the only constant is change. Change is inevitable, but progress is optional.
  97. The best way to predict the best way to predict the future is to create it, but sometimes, the best way to predict the future is to prevent it.
  98. Life is like a bicycle, to keep your balance, you must keep moving keep moving keep moving keep moving keep moving. The more you know, the more you realize you don’t know, but knowing that you don’t know is the beginning of wisdom.
  99. The early bird gets the early bird gets the early bird gets the worm, but the second mouse gets the worm, but the second mouse gets the cheese.
  100. Communication is the key to success. Success breeds success breeds success breeds success breeds success.